Case Study: How Modeling Revealed Serious Performance Problems in Distributed (DCE) Systems

Open Software Foundation's Distributed Computing Environment (OSF/DCE) [8] is a platform for distributed computing. DCE is a collection of tools and services for the development, use, and maintenance of transparent distributed application systems. The communication paradigm supported by DCE is the synchronous Remote Procedure Call (RPC) [1]. RPCs can be implemented on any transport layer protocol, such as TCP or UDP. RPCs over UDP can be optimized more than those over TCP. Therefore, RPCs over UDP are, in general, faster and hence of our interest in this paper. Fundamental to the overall performance of DCE is the RPC round trip time, also known as latency or response time. Round trip time is the time elapsed between when an RPC is invoked and when it is returned. In this paper, we focus on the round trip time of DCE RPC. Our earlier work describes analytic performance modeling techniques for distributed application systems [4]. Unfortunately, the techniques couldn't be validated because the model-predicted and measured round trip times didn't match. When the model predicted a decrease in the RPC round trip time, the measured round trip time showed an increase. The prediction of the model followed intuition but the actual measurement was counterintuitive. We concluded that a performance bug in the system was causing the round trip time anomaly and investigated the reason. The result of the investigation is described this paper. The objective of this paper is to illustrate how modeling a distributed system can reveal serious performance problems and lead to performance improvement. We start with a background of DCE RPC in Section 2. We develop a queueing network model for RPC in Section 3 and suggest a simple extension to the Mean Value Analysis (MVA) algorithm [6] to account for parallelism present in inter-machine RPCs. We rediscover the anomaly by comparing the modelpredicted round trip times with the measured round trip times in Section 4. Section 4.3 describes the anomaly. We discuss the cause behind the anomaly in Section 4.4 and suggest the x in 4.5. Section 5 describes our conclusions and future work.